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FINAL  REPORT 


This  is  a final  report  of  the  work  con^pleted  under 


AFOSR  77-3286.  In  Section  I 


reviewed  and  in  Section  II  the  work  completed  under  the 


grant  is  discussed.  Finally,  in  Section  III  work  in 


progress  under  the  support  of  AFOSR  77-3286B  is  discussed 
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1.  Kesearch  Proposal. 

Problems  in  Optimal  Nonlinear  Filteriny 

In  this  part  of  the  researcli  wa  seek  a computationally 
convenient  technique  for  solving  tlie  filtciing  and  prediction 
probleiiis  for  a class  of  nonlinear  stochastic  differential 
equations  subject  to  partial  observations  at  discrete  time 
points.  Tlie  applications  of  the  technique  described  below  to 
trajectory  estimation  are  apparent. 

Let  there  be  given  q functions  f^  which  are 

known  to  us.  Suppose  for  some  ic{l,2,...  q),  unknown  to  us, 
the  state  of  the  process  is  evolving  according  to  the  vector 
stochastic  differential  equations 

dx  = f^(t,x)dt  + g(t,x)dw,  x(o)  = x. 

At  the  discrete  times  t^ , . . . t^,  which  are  known  in  advance, 
an  intermediate  observer  (usually  a machine)  receives  noise 
corrupted  observations  of  tlie  stochastic  process  x(t).  Tlicse 
observations  y(t)  satisfy  the  stochastic  differential  equations 

dy  = H(t,x,y)dt  + g(t,x,y)dw,  y(o)  = 0. 

Suppose  that  y(t)  t r"*  for  some  m.  Assume  there  exists  known 
pairwise  disjoint  sets  i = l,...,m,  L' = R*''-  Tlien  at  eacli 

time  t j / j *=  1,2,...  p,  we  receive  from  the  intermediate  observer 
only  the  information  as  to  which  of  the  events  y(tj)  ^ has 
occurred. 

Our  first  problem  is  to  determine  for  eacli  function  f^^, 


the  probability  that  the  function  f^  is  beiny  used  given 
available  inforraation. 

Our  second  prediction  problem  is  the  following:  Given  our 
information  at  tiraes  t » t^^,  t2»  • • • » t^,  and  t he  function  f^, 
determine  the  best  prediction  in  mean  square  o^  some  function 
h(x(T))  of  the  process,  T ^ t^. 

Let  us  discuss  problera  2 which  we  have  recently  solved 
theoretically.  For  each  possible  information  set,  we  must 
solve  a coupled  set  of  second  order  partial  differential 
equations.  Since  the  boundary  conditions  are  of  Cauchy  typo, 
they  can  be  solved  nunierically . There  are  la'^p  information 
sets,  each  requiring  the  solving  of  the  cr'upled  sot  of  partial 
differential  equations.  Although  these  problems  appear  complex, 
they  have  the  important  practical  advantage  that  they  can  be 
completely  preconiputed  and  do  not  need  to  bo  solved  in  real-time. 

In  this  research  we  intend  to  consider  the  problems  discussed 
above  looking  at  both  the  theoretical  solution  and  effective 
computational  methods  that  can  be  developed  froju  the  theoretical 
solutions . 

APPROXIMATION  TECHNIQUES  IN  STOCHASTIC  CONTROL  PROBLEMS 

In  this  researcl)  we  continue  our  investigation  of  approxi- 
mation techniques  for  a wide  class  of  discrete  and  continuous 
time  stochastic  control  probletas.  This  research  centers  on  small 
noise  problems  in  the  continuous  case  and  on  nonclassical 
stationary  problen\s  in  the  discrete  case.  Emphasis  is  placed 
on  the  development  and  theoretical  justification  of  techniques 
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wiiich  yield  computationally  tractable  al<tori  that  answer  the 

following : 

(1)  approximations  to  the  opt  baa 1 cost  ami  the  cost  ol 
using  a particular  control . 

(2)  approximations  to  the  optiia.'i  control. 

(3)  evaluation  of  the  relative  poi  tonaanc>-  ot  two  ct>ntrols. 

(4)  estimates  for  the  deterioration  in  system  performance 
due  to  tlie  failure  to  observe  ci?rtain  r.vstem  ctmiponents. 

SMALL  NOISE  PROBLEMS  IN  CONTINUOUS  TIME 
Outline: 

One  general  problem  has  been  to  solvi  "appi oximately"  the 
stochastic  control  probld^i  in  tonus  of  quantities  computable  1 rom 
the  solution  to  the  corresponding  deterministic  control  problem 
when  the  noise  entering  the  system  equation  is  a "si.iall"  pvtroiueter. 
The  general  procedure  is  to  theoretically  establisJt  expvUisions  of 
the  optiiaal  cost  and  control  in  powers  of  tl\e  noise  coefficient. 

The  expansions  then  suggest  appropriate  forms  for  suboptimal 
controls  and  numerical  techniques  for  detenuining  them.  Tlie 
theoretical  development  of  the  expansions  involves  the  interplay 
between  probability  and  partial  differential  equations.  See  [41 
for  a general  review  of  tlu  current  state  in  this  area. 

The  theoretical  portion  of  this  approach  for  the  coiupletely 
observable  problem  was  treated  successfully  by  Fleming  [3] ; the 
numerical  algoritlin:s  suggested  by  the  theoreius  were  developed 
and  employed  on  some  two  dimensional  examples  by  this  author  in 
(51.  A partial  solution  to  the  open  loop  control  problciu  was 


S 
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developed  by  the  author  in  [6]  and  [8] . In  this  research  wc 
intend  to  extend  the  results  of  the  open  loop  control  problem 
and  derive  similar  results  for  the  sampled- data  problem. 

Finally r we  intend  to  investigate  the  .‘  tat ionary  small  noise 
control  problem  in  the  case  of  both  cc"pl(.'t  e and  partial  obstr- 
vations.  Under  certain  assiuuptions  we  liavo  ciin,  ut-'d  an  c'lpansion 
of  the  cost  of  using  a fixed  control  in  I M . The  expansions  in 
this  case  are  nice  in  that  the  coef  f ic.i  eni  s ol  the  expansion 
satisfy  algebraic  equations  and  hence  can  lo  computed  easily. 

In  this  research  we  seek  expansions  of  tJie  optimal  cost  and 
control  under  suitable  assumptions.  Further  we  intend  to  investi- 
gate approximation  techniques  suggeste  ! In  our  already  established 
result. 

Details : 

Many  problems  of  continuous  time  stochastic  control  can  be 
fonaulated  as  follow.  Let  x(t)  = {x^(t),  X2  ( t)  , . . . ,Xj^(  t)  ) t 
denote  the  state  of  the  systcin  at  time  t,  and  suppose  tliat  the 
states  evolve  according  to  the  stochastic  diffeiential  equations 


dx(t)  - f(t,x(t),u(t))dt  + b(t,x(t))dw(t),  t ^ tp,  (1) 

with  initial  condition  “ ^0  where  Xq  is  a constant 

and  B is  defined  below,  in  (1)  w is  a normalized  n dimensional 
brownian  motion  process  and  u(t),  whicli  takes  values  in  a compact 

If 

set  K c.  R , is  the  control  applied  at  time  t.  As  is  custoiaary 
dependence  on  (1)  and  other  equations  on  the  point  m in  the 
sample  space  n is  omitted. 
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Finite  time  control  problems.  Let  t denote  the  first  time 
t ^ tp  such  that  (t,x(t))  X Q where  Q = (tg^T)  x B and  B 
is  an  open  subset  of  r”  such  that  its  boundary  9B  is  a 
siaooth  manifold  with  compact  closure.  Then  the  optimal  control 
problem  is  to  choose  m k.  the  class  of  adraissible  controls, 
so  as  to  minimize  the  quantity 


E 


f L(t,x(t)/ 


t 


0 


u ( t)  ) dt 


(2) 


where  E denotes  expected  value. 

The  class  % is  chosen  to  reflect  the  an\ount  of  information 
available  to  the  controller.  In  this  part  of  the  research  we 
consider  three  special  cases: 


Open  loop  case.  No  observations  of  x(t)  are  made  for 
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Define  (tQ,XQ)  , i|;^(tQ,XQ),  x^(tg,XQ)/  and  <t>°(tQ,XQ) 
to  be  the  optimal  cost  for  the  point  (tQ/X^)  in  the  completely 
observable,  open  loop,  sarapled-data,  and  deterministic  control 
problons,  respectively,  with  the  superscript  e indicating  that 
b(t,x)  = is  used  in  (1).  Assume  that  (tQ,XQ)  € N 

where  N is  a region  of  strong  regularity.  A region  of  strong 
regularity,  see  [3],  p.  480,  guarantees  that  the  optimal  de- 
terministic control  and  trajectory  have  certain  regularity 
properties.  Then  Fleming  showed  in  [3]  that 


4»^(tg,XQ)  = (t)°(tQ,XQ)  + eO^  + o(e)  (3) 

and  this  author  showed  in  [6]  in  case  B = that 

(J;^(tQ,XQ)  = 4i°(tQ,XQ)  + e02  + ©(e)  (4) 

where  0^^  and  ©2  can  be  determined  from  a knowledge  of  the 
solution  of  the  corresponding  deterministic  control  problem. 

In  this  research  the  author  seeks  to  extend  the  result  (4)  when 
B merely  satisfies  the  earlier  hypothesis  and  also  to  establish 
the  analagous  result 

X^(tg,XQ)  = (J)°(tQ,XQ)  + £©2  + o(£)  (5) 

in  the  sampled-data  problem. 

Let  y°(t,x)  denote  the  optimal  feedback  control  in  the 
completely  observable  case,  then  Fleming  [3]  showed  that 

Y^(t,x)  = Y^(t,x)  + eW(t,x)  + o(e)  (6) 
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uniformly  in  (t,x)  on  compact  subsets  of  tliose  'uyions  in  N 
where  Y is  a C function.  We  seek  siuiilai  o.v  aiisions  of 
the  optimal  control  in  the  open  loop  and  sarpl.-d-oata  cases. 

Under  assumptions  which  included  that  each  i'p«  n lot  p ocnlrol 
generate  a Gaussian  process,  we  e.stablisheii  n i8I  C 
expansions  of  the  optimal  control  in  powers  of 

The  numerical  methods  f tr  v'  loul.^tinu  the  ij.i  ’n*  i ties  0^ 
and  W(t,x)  in  (3)  and  (6)  has  been  discussed  la  a t>ape  15] 
by  the  author:  the  author  has  also  developed  under  certain 
assumptions  a numerical  metluxi  for  finding  "b<  .,l"  controls  >>1 
the  form  u^  + : r for  some  functien  /,  ii  the  open  1 nii  cas  . 

In  case  an  expansion  of  the  open  loop  cent  ol  is  es tabl  islievi , 
then  the  function  z must  be  the  cooffiv-i  lu  t)f  i . 

Stationary  control  problei.is.  Let  ''Jf  deiu  i o the  class  o.  a..'  <'- 
nomous  feedback  controls  for  whrcli  tlie  autonomous  version  i ' tl) 
has  a unique  ergotlic  measure.  See  Kushner  [11],  Wonham  (14J, 
and  Zakai  116)  for  a discussion  of  this  asrumption.  The  orgou  c 
measure  depends  on  both  u t 'i'  and  e will  be  denoted  by 
Vi(u,t:,*).  The  optintization  probldii  is  to  elioose  u t '4'  so  that 

J(u,e)  = I L (x  ,u  (x)  ) du  (u,  i- ,x)  (7) 

r" 

is  minimized.  Denote  the  miniriium  cost  by  J*(i  ) and  the  optimal 
control  by  \x^ . Under  certain  assumptions  we  have  shown  17]  that 
for  each  u, 

J(u,e)  “ I + oCt") 

k«0 


(8) 
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for  sotne  constants  0j^,G2*  . . . .0^^  which  satisfy  uiyebraic 
equations.  The  intesjer  n depunUs  upon  the  suioothnesa  oi  i,L, 
and  u. 

In  this  research  we  seek  expansions  of  J*(«  ) and  u'  in 
powers  of  t.  We  will  also  examine  the  utility  of  iiupieJ.ientiny 
a truncated  expansion  of  (8)  as  an  approx iuvation  to  the  cost  in 
order  to  determine  suboptmial  controls.  Our  pxesont  researcii 
seetAS  to  indicate  that  this  is  a good  techniqvje  when  the  class 
of  controls  is  restricted  to  those  liaviny  prescribed  forms 
with  unknown  paraiueters. 

The  stationary  control  probleia  has  bet-n  previously  stuiUed 
by  Wonham  and  Caslmvan  [15]  who  used  stvitiscicai  I ineari  aa  tiun 
as  an  approx ii.iation  tool. 

STATIONARY  DISCRETE  TIME  CONTROL 

Discrete  time  stochastic  control  probieius  h.ive  been  tlr'vouyhly 
considered,  however,  most  prior  work  considers  the  case  in  whicli 
the  controller  reraembers  all  previously  obtaintnl  information. 

This  has  been  called  the  classical  inforntation  pattern  by 
Witsenhausen  [13]  . We  intend  to  investiyate  the  statioiuiry 
control  case  where  the  controller  has  only  partial  observations 
of  the  systeia  state  and  no  memory.  This  eiuphasis  is  motivated  by 
systems  in  which  is  too  difficult  or  expensive  to  observe  all 
system  components,  and  systems  in  which  it  is  difficult  to 
implement  controls  using  past  information. 

Under  certain  reasonable  assume  tions  we  have  succeeded  in 
reducing  the  optimization  problem  to  a probleitt  in  nonlinear 
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progrontming  (9)  . We  liave  also  constructed  exoiuples  to  sliow  that 
one  can  do  better  by  using  randuitized  controls  then  by  oj>ly  using 
nonrandonvized  controls  of  the  current  observed  data.  In  problojus 
of  this  type  with  ccapiete  observations  it  is  known  that  the 
optiraization  probltHi\  can  be  treated  as  a probliM  in  linear  i)ro- 
grar.mving  [12]  and  that  the  controller  cannot  do  hotter  by 
considering  randomized  controls  (IJ. 

In  this  research  the  author  intends  to  investigate  the 
cauputational  complexity  of  tl)o  nonlinoiir  progr vuaming  algor itluu 
ajxl  to  seek  conditions  on  the  partially  observable  probieui  wiiich 
guarantee  that  the  optimal  control  can  be  chosen  to  be  nun- 
randomized.  This  latter  question  is  important  Ix'tli  lor  designing 
cor.iputational  aigoritluas  and  systeiti  design. 

The  importance  of  such  problems  was  first  brovight  to  our 
attention  by  Dr.  Raymond  Rishel. 
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II.  Work  Completed  Under  the  Grant. 

(1)  Stochastically  perturbed  limit  cycles,  J.  Applied  Probability 
15,  311-320  (1978) . 

An  important  question  in  the  stability  and  control  oC 
stochastic  systen\s  is  the  determination  of  the  limitincj  long 
time  behavior  of  tlie  system  using  a fixed  control.  This  work 
answered  that  question  in  the  case  of  a stochastic  system  perturbed 
by  a small  additive  noise  term  where  tJie  control  is  such  that  the 
corresponding  deterministic  systtai^  possesses  a stable  limit  cycle. 

It  is  shown  that  in  tlie  limit  of  large  tbiie  the  stochastic 
systen*  is  near  the  limit  cycle.  T)iis  is  a stability  result. 

Moreover,  one  can  compute  approximately  at  which  portions  of  the 
limit  cycle  one  is  most  likely  to  be  found.  Further  various 
stationary  average  can  be  computed. 

These  results  will  be  of  use  in  designing  approxbuate  controls 
for  stationary  stocliastic  control  systems.  For  a detailed  dis- 
cussion of  these  results,  see  the  completed  above  paper. 

(2)  A Minimum  Principle  for  the  Principal  Eigenvalue  for  Second- 
Order  Linear  Elliptic  Equations  Witli  Naturvil  Boundary  Conditions, 
Conununications  in  Pure  and  Applied  Mathematics  31(1978)  509-519. 

This  work  gives  a new  characterization  of  the  principal  eigen- 
value for  second-order  linear  elliptic  partial  differential 
equations,  not  necessarily  solf-ad;ioint , with  both  natural  and 
Dirichlet  boundary  conditions,  and  also  gives  a new  alternative 
numerical  method  for  calculating  botn  tlie  principal  eigenvalue  and 
corresponding  eigenvector  in  the  case  of  natural  boundary  conditions. 
The  principal  eigenvalue,  if  appropriate  sign  changes  are  made, 
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determines  the  stability  of  equilibrium  solutions  to  certain 
second  order  nonlinear  partial  differential  equations.  The 
corresponding  eigenvector  enables  one  to  di'tuimine  the  first 
approximation  of  the  solution  of  the  nonlinea  •. qu.ttion  to 
variations  of  the  initial  condition  from  the  jui librium 
solution.  These  nonlinear  equations  are  important  in  the  appli- 
cations. For  these  reasons  it  is  important  to  have  tliesc 
characterizations  of  the  principal  eigenvalue  aiiu  •igen>'ector . 

Our  method  converts  the  detem^ination  oi'  the  eigenvalae 
and  eigenvector  to  determining  the  solution  of  a stationary 
stochastic  control  problem.  This  latter  prc  blem  is  solved  and 
from  it  a numerical  schane  arises  naturally.  This  metliod  appears 
to  have  applications  in  solving  other  proLlems. 
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III.  Work  in  Progress. 

In  the  recent  work  in  (2)  above,  we  were  able  to  derive  a 
new  characterization  of  the  principal  eigenvalue  for  second  order 
linear  elliptic  partial  differential  equations,  not  necessarily 
self-adjoint,  with  both  natural  and  Dirichlet  boundary  conditions, 
and  also  give  a new  alternative  numerical  method  for  calculating 
both  the  principal  eigenvalue  and  corresponding  eigenvector  in 
the  case  of  natural  boundary  conditions. 

We  shall  use  the  above  results  to  determine  the  asyraptotic 
behavior  of  the  principal  eigenvalue  for  some  singularly  perturbed 
eigenvalue  problems  as  a small  nuisance  parameter  tends  to  zero. 
The  principal  eigenvalue  is  the  optimal  value  for  a singularly 
perturbed  stationary  stochastic  control  problem.  VJe  are  thus 
able  to  determine  the  asymptotic  behavior  of  the  optimal  value 
of  certain  stationary  stochastic  control  problems. 
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